Strain-sensing goniometers, systems and recognition algorithms

ABSTRACT

Goniometers are provided having internal compensation employing two opposing variable resistance strain sensing elements separated by a flexible film. The goniometers may be used in sets for detecting complex hinge or joint movements, where patterns of hinge or joint positions may be used to define symbols. Algorithms are provided for parsing values from a system of goniometers in motion having information content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Divisional of application Ser. No. 08/148,991filed Nov. 8, 1993, now U.S. Pat. No. 5,442,729 issued Aug. 15, 1995;which is a Divisional of application Ser. No. 07/625,305 filed Dec. 10,1990, now U.S. Pat. No. 5,280,265; which is a Continuation ofapplication Ser. No. 07/258,204, filed Oct. 14, 1988, now U.S. Pat. No.5,047,952 issued Sep. 10, 1991; each of which disclosures isincorporated in its entirety herein.

INTRODUCTION

Background

This invention relates to systems which sense spatial relationships andto the individual sensors used in the systems. The system is capable ofproducing a decision based on the values of one or more sensors. Thesystem finds application in a variety of gesture recognitionapplications, exemplified by communication devices for nonvocal deaf ordeaf-blind individuals. The system also finds application providinginformation on complex animate and inanimate motion which can be storedand/or processed immediately to construct a true or transformed model ofthe original motion. The model may be used to analyze the originalmotion or used as a control input to a second application.

There are various methods of sensing spatial relationships ofinterrelated parts, such as human and robotic joints and other moveableparts. In particular, there is a strong interest in improving on anddeveloping new goniometric, i.e., angle measuring, devices. Commonlyused goniometers, such as optical encoders, resolvers, potentiometers,Hall-effect sensors, and the like, are based on optical, electrical,mechanical and electromechanical technologies, etc. Most availablegoniometers are housed in bulky mechanical structures which can bedifficult to mount and often interfere with the motion they are intendedto track. The devices can be expensive, have limited resolution and canbe difficult to use in a case where the bend axis is not rigidly definedand varies during flexure (e.g., a finger joint).

There is therefore a need for a low-profile, high resolution goniometerwhich is easily and unobtrusively mounted to a surface to monitor anglesbetween two or more moveable structures associated with that surface(e.g., to monitor the bending of a joint or hinge, or the interrelationbetween the angles of a number of joints or hinges). The output of sucha goniometer may be used to control the movement of computer generatedobjects or figures on a computer monitor, as well as to control theoperation of physical machinery, and the like.

Relevant Literature

U.S. Pat. No. 4,414,537 describes an instrumented glove which usesoptical bend sensors, while U.S. Pat. Nos. 4,414,537 and 4,542,291describe an instrumented glove using related optical sensing technology.Instrumented gloves are described by Foley, "Interfaces for AdvancedComputing," Scientific American, October, 1987, pp. 127-135. A straingage flex sensor is described in Jespersen, et al., "Joint AnglePosition Sensor," 40th ACEMB, Niagara Falls, N.Y., Sep. 10-12, 1987, p.104. A Hall-effect sensor used to measure finger joints is described inMarcus, et al., "Sensing Human Hand Motions For Controlling DexterousRobots," 2nd Annual Space Operations Automation and Robotics Workshop,Wright State Univ, Jul. 20-23, 1988.

SUMMARY OF THE INVENTION

Sensors and sensor arrays are provided where each sensor ischaracterized by having at least one flexible variable resistance strainsensing element mounted to a flexible backing. In one embodiment thesensor motion is restricted by a guiding element and at least one end ofthe sensor is able to slide relative to the guiding element material.The guiding element is affixed to a surface and the sensor is guided bythe guiding element to produce an electrical signal representative ofthe angle between two structures related to the surface. A plurality ofsensors can be used to measure the conformation of the surface at aplurality of points.

Adaptive finger-gesture recognition algorithms are also provided wherevarious predefined hand formations are recognized from continuouslyvarying finger motion. Both a Bayesian and an artificial neural networkclassifier are described which select the appropriate gesture inconjunction with a dynamic gesture parsing routine.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of a communication system inaccordance with the invention.

FIGS. 2A and 2B illustrate a hearing person and a deaf-blind personcommunicating with a communication system in accordance with theinvention.

FIG. 3 is a plan view of an instrumented glove in accordance with oneembodiment of the invention.

FIG. 4 is a side view of a finger joint and a strain gage flex sensor asused in the glove of FIG. 3.

FIG. 5 is a functional block diagram schematic of a strain gagemultiplexing bridge circuit for providing signals to the computer in thecommunication system of the invention.

FIG. 6 is a hand space diagram for a three-dimensional spaceillustrating a hand state vector and three letter beacons as identifiedby a letter recognition algorithm in accordance with a feature of theinvention.

FIG. 7 is a functional block diagram of a prototype system developed atStanford University.

FIGS. 8A and 8B are a flow diagram of software for implementing a letterrecognition algorithm in the system of FIG. 7.

FIG. 9 is a flow diagram of a Wait to Exit recognition ball routine inthe flow diagram of FIGS. 8A and 8B.

FIG. 10 is a flow diagram of a Check for Inputs routine in the flowdiagram of FIGS. 8A and 8B.

FIG. 11 is a flow diagram of an I/O and sampling software routine in thesystem of FIG. 7.

FIG. 12 shows views of the variable-resistance strain-sensing sensor.

FIG. 13 is a perspective view of a sensing array where one sensingelement is not positioned parallel to the other.

FIGS. 14A and 14B show one sensing element arrangement, and 14C and 14Dshow a second sensing element arrangement.

FIG. 15A shows the sensor signal processed by analog circuitry for useby an analog data utilization device. FIG. 15B shows the sensor signalprocessed by analog and digital circuitry for use by a digital datautilization device.

FIG. 16A shows the sensor signal controlling a robot gripper. FIG. 16Bshows the sensor signal controlling a graphic image.

FIG. 17 is an application for the subject sensor in measuring surfacecontours.

FIG. 18 shows a sensor in one type of guiding element.

FIG. 19 shows a sensor in a guiding pocket.

FIG. 20 shows a sensor guided by a channel.

FIG. 21 shows the subject sensors located at desirable locations on thehand.

FIGS. 22A and 22B are two gloves with portions of the gloves removed orreplaced with mesh.

FIGS. 23A-23C are three techniques for using the subject sensor tomeasure joint abduction.

FIGS. 24A and 24B are sensors shaped in a "U" to measure jointabduction.

FIGS. 25A-25D are bend-limiting elements.

FIG. 26 is a glove where the fingertip material is removed.

FIGS. 27A and 27B show the subject sensor located at various locationson a body.

FIG. 28 shows a sensor attached to a hinged structure.

FIG. 29 shows a sensor attached to an alternative hinged structure.

FIGS. 30A and 30B show the subject sensor used to simultaneously measureangle and tension.

FIG. 31 shows the sensor measuring the position of a pulley belt.

FIG. 32 shows the sensor measuring the bend of a cantilever specimen.

FIG. 33 shows the sensor measuring axial rotation.

FIG. 34A shows an instrumented glove being used with various additionalperipheral and communication devices in a communication system, where agesture recognition algorithm is used capable of producing symbolicoutput and control signals, including synthesized speech. FIG. 34Bprovides the dorsal view of an instrumented glove and FIG. 34C providesthe palmar view of the glove.

FIGS. 35A-35C show three views of a configuration for measuring jointabduction.

FIG. 36 shows beacons in hand space, with recognition and unrecognitionhyperellipsoids.

FIG. 37 is a block diagram showing a gesture parsing routine andclassification routine which are part of an overall recognitionalgorithm.

FIG. 38 is a plot of an example hand state velocity vector magnitudeversus time curve with various points of interest marked.

FIG. 39A is a feed-forward neural network. FIG. 39B is a sigmoidalsquashing function.

FIGS. 40A and 40B are two example sets of clusters of labeled trainingdata points and corresponding TSNNC decision trees.

FIGS. 41A and 41B are block diagrams of the TSNNC training andclassification algorithms respectively.

FIG. 42 is a plot of an example hand state velocity vector magnitudeversus time.

FIG. 43 is a plot of an example hand state acceleration vector versustime.

FIG. 44 is an example 2-D curve of hand state trajectory.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Sensor and sensor arrays are provided where each sensor is characterizedby having a variable resistance strain sensing element (e.g., a straingage) mounted to a flexible backing (FIG. 12). The sensor arrayscomprise one or more (usually not more than two) variable resistancestrain sensing elements. Multiple sensing elements are preferablymounted on opposite sides of the flexible backing. When mounted onopposing sides, the sensing elements are preferably positioned parallelto each other, but may also be positioned orthogonally (FIG. 13) or atother arbitrary angles.

Each sensing element consists of an electrically conductive materialwhose resistance varies with strain, such as metals and metal alloys,conductive inks (e.g., an adhesive impregnated with electricallyconducting particles or fibers), semiconducting materials, conductiveelastomers (e.g., an elastomer impregnated with electrically conductiveparticles or fibers), organic conductors, and the like. The materialemployed and the design of the sensor array preferably provides for asubstantially linear response to changes in angle. The electricallyconducting material is generally mounted to the flexible backing in agrid pattern, where the majority of the material is in strips along theperpendicular to the primary bend axis, thus, the sensor is relativelyinsensitive to bending along axes other than the primary axis.

The electrically conductive material for the strain sensing element isadhered to the electrically non-conductive flexible backing by adhesive,by casting the backing onto the electrically conductive material, byvapor deposition of the electrically conductive material onto thebacking, or by other convenient means. The electrically conductivematerial is generally present as a thin film or foil strip, convenientlyless than about two mils thick, but usually at least about 0.05 mil. Thelength of the sensing element will generally vary with the application,usually not less than one-quarter inch and not exceeding one foot. Forexample, the length of the sensing element ranges from about one-half totwo inches for a glove or from about eight to ten inches for an elbow orknee. The width of the sensing grid will usually not exceed one-halfinch, being about one-eighth inch for a glove and one-quarter tothree-eighths inch for an elbow or knee sensor.

The backing material is conveniently an inert, low thermal expansion,flexible, electrically insulating, low mechanical hysteresis (i.e., dueto bending) material, which may be a film, woven fabric, insulated metalor other supporting substrate with the appropriate physical andelectrical characteristics. For the most part, plastic films find use,particularly films which may be cast. Films which may find use includepolyimide, mylar, polypropylene, polyester and the like. The backingmaterial will generally be of a thickness in the range of 0.2 to 10 mil,preferably from about 0.2 to 2 mil. With a thickness of an annealednickel-copper electrically conductive material of about 0.1 mil, apolyimide backing of 0.5 mil is preferred for a glove.

When a single sensing element is adhered to a flexible backing, thebacking is inelastic enough along the length dimension compared to theelasticity of the sensing element so that the sensing element isstrained when the entire sensor assembly is flexed. When a sensingelement is mounted to each side of a backing the relative elasticityalong the length dimension of the backing is less critical. As long asduring flexure of the entire sensor assembly the compression inthickness of the backing is negligible, the structure remains symmetricin cross-section and the neutral bend axis passes down the middle of thebacking, thus one sensing element sees strain of equal magnitude butopposite sign to the other sensing element. This differential strainsignal is ideal when used as input to a differential amplifier since notonly is the desired signal resulting from bend of arbitrary angledoubled, but error terms introduced into the strain signal due tonon-ideal or parasitic effects (e.g., change in temperature, electricalnoise, mechanical hysteresis, changes in material properties due toaging or other ambient influences) are present in the signals from bothstrain sensing elements and thus their effects are reduced due to thecommon mode rejection of the differential amplifier.

A number of individual sensors or an array of sensors may be fabricatedtogether by affixing the electrically conductive strain sensing elementto both sides of a piece of backing material and then etching both sidessimultaneously to yield paired grids, i.e., one on each side of thebacking material, where each pair of grids makes up an individualsensor. Similarly, a sensing element may be affixed to one side of apiece of backing material and then etched. Two mirror image etchings ofthe same array of sensors on separated backing sheets can be adheredtogether to produce grid pairs. When appropriately matched sheets ofgrid arrays are used, the two sheets may either be adhered backingmaterial-to-backing material (FIGS. 14A and 14B) or backingmaterial-to-grid side (FIGS. 14C and 14D). The latter would produce athinner total backing thickness between the two matched grids. Thismethod is used when production techniques makes it impossible to make athinner single backing sheet. When both backings are adhered to eachother a more symmetric structure is produced at the expense of doublingthe total effective backing thickness.

The maximum strain seen by each sensor is adjusted by varying thethickness and elastic modulus of the plastic backing to which the sensoris mounted. The backing thickness and elastic modulus of the backing towhich the foil is mounted. The sensors are preferably wired in aWheatstone bridge configuration that employs an analog multiplexer toselect a particular strain sensor bridge voltage (FIG. 5). The analogsignal from the sensor can be processed by analog circuitry (FIG. 15A)and used by an analog data utilization device, which may, for example,use the analog input in applications such as motor control (e.g., speedand position), control of an artificial limb (e.g., position and motioncontrol), telerobotics (e.g., control of a remote robot manipulator orgripper, FIG. 16A), control of audio amplifiers (e.g., volume and tone),and the like. The analog signal can be converted to a digital signal forinput to a computer or other digital data utilization device (FIG. 15B).Applications include, in addition to the analog applications mentionedabove, positioning of a cursor (e.g., for use in CAD design or textediting), construction of a true or transformed graphic model (FIG. 16B)of the original motion (e.g., for use in Virtual Reality), recording ofmotion data for analysis and comparison (e.g., studying of human motionand flexibility), and the like. The transformed graphic model may takethe form of a puppet, animated body part, animated character, animatedform (e.g., a life form) and the like.

The total thickness of the entire sensor assembly is in the range ofabout 0.2 to 12 mil, where the total thickness of the annealednickel-copper/polyimide sensor for an instrumented glove ranges fromabout 0.3 mil to 2 mil. The dimensions of the sensor and the nature ofthe materials used should allow the sensor to follow an angle of bendingto preferably at least about ±115 degrees.

The electrically conductive strain sensing element of the sensorassembly is connected to amplification and data processing circuitry byelectrically conducting means, such as lead wires, electrical traces(etched or deposited), and the like. For typical applications requiringone or more portions of the body to be instrumented, typical leadwiresrange from 26 to 36 AWG, are comprised of stranded wire and have aflexible insulation. For an instrumented glove, 36 AWG stranded copperTeflon-insulated wire may be used.

There are many applications for the subject sensors where a singlesensor may be employed or groups of sensors may be combined and appliedin various manners. Included among the applications are measuring body(e.g., finger) movements, bending of hinges, travel of conveyor ortiming belts and other flexible surfaces, contoured surfaces, angle ofbend of cantilevered structures, axial rotation and the like.

In using the subject sensors for following body or surface contours(FIG. 17), a film, fabric or other material is employed which closelyfollows the flexible joint or surface to be measured. With a hand, thefabric may be formed in the form of a partial or whole glove. With otherparts of the body, the fabric may be formed as a knee or ankle brace,vest, suit or the like. The fabric will conform to the contour of thesurface and hold the sensing element in close juxtaposition (FIG. 4) tothe surface making the sensor able to measure changes in theconformation of the surface. The sensor may be mounted onto or over thesurface in a guiding element (FIGS. 4, 18, 19, 20). The guiding elementmay take the form of a pocket in which the sensor may reside (FIGS. 18,19). The sensor may freely move within the constraints of the guidingelement, may be affixed at one end with the other end free to move (FIG.20), may be affixed at both ends with slack in the sensor between theends, or may be affixed to the glove material where the glove materialis elastic and stretches when the sensor flexes (e.g., the sensors maybe molded directly into a rubber or plastic material).

Of particular use for the sensors is the use of a plurality of sensorsin a glove (FIGS. 3, 21, 22) for providing electrical signals related tomovement of the hand and fingers. The glove is equipped with sensors asdescribed above, which monitor the angles of the finger joints. At leasta portion of the sensor resides in a guiding channel to guide the sensorrelative to the joint as the joint is flexed. For example, the guidingchannel may comprise one or more loops through which the sensor passes(FIG. 20) or the sensor may be completely enclosed in a pocket andaffixed over each joint to be monitored (FIGS. 18, 19).

The sensors can be placed at various locations in relation to the jointsof the hand, for example by affixing the guiding channels or pocketsonto a glove or sewing the pocket directly into or onto the glove.Typical bend sensor locations are over the metacarpophalangeal, proximaland distal interphalangeal joints of the four medial fingers andtrapeziometacarpal, metacarpophalangeal and interphalangeal joints ofthe thumb (FIG. 21).

Abduction sensors may be used and in particular are used to measure theabduction of the metacarpophalangeal joints of the four fingers and thetrapeziometacarpal joint of the thumb. As with the sensors in general,the abduction sensors may reside in guiding pockets, may be free toslide, affixed at one or both ends, or be affixed to an elastic glovematerial (e.g., molded into a rubber or plastic material). "Absolute"abduction sensors may be located on the side of the index and smallfingers (FIGS. 21, 23B, 23C) or positioned on edge on the dorsal side ofthe hand near the abducting joint (FIGS. 23A, 35A-35C). "Relative"abduction sensors may be located between any neighboring pair of digits,particularly near the metacarpophalangeal joints to measure the anglebetween the digits. The relative abduction sensor may take on a U-shapedconfiguration and may be positioned between two digits in the abductionplane (FIG. 24A) or located on the dorsal surface of the hand (FIG. 24B)above the abduction plane. When a U-shaped relative abduction sensor islocated in the abduction plane a bend-limiting element is preferablyplaced between the joints near the sensor to prevent the sensor frombeing bent beyond the minimum bend radius (and from being creased). Thebend-limiting element may take the form of a tube (FIG. 25A), half tube(trough, FIG. 25B), flexible structure which suddenly binds and becomesstiff at the desired bend radius (FIGS. 25C, 25D), and the like.Additional sensors measure the rotation of the thumb across the palm andthe arch of the metacarpus near the small finger. FIG. 21 shows sensorson the back of the metacarpus to measure the thumb rotation andmetacarpal arch near the small finger. Sensors may also be placed on thetop and side of the wrist to measure pitch and yaw of the wrist inrelation to the forearm (FIG. 21).

The fingertips of the glove are preferably removed (FIGS. 22, 26) topermit the user to reliably grasp objects when it is not necessary tomeasure the flexure of the outer finger joint (distal interphalangealjoint). Material covering any knuckle on the palm side of the glove, aswell as a large portion of material covering the palm, may also beremoved to prevent binding and provide ventilation (FIG. 22A). Elasticmesh material (FIG. 22B) may be used to replace portions of the fabricto provide greater comfort and ventilation and to help decouple onesensor from another due to tension through the fabric from movement ofan adjacent joint.

For other body parts, the bend and abduction sensors will be employed ina manner similar to the glove embodiment. The sensors are long enough tospan the entire arc subtended by the angle being measured. The guidingpocket may be an integral part of or affixed to a tubular elastichousing which is placed around the joint or region to be monitored.Areas which may be monitored include ankle, waist, elbow, back, neck,wrist, shoulder, hip, groin, knee, or other flexible region (FIGS. 27A,27B).

Sensing applications for other than body parts allow for variations instructure from the measurement of body parts. For example, with hingestructures having one or more hinges along the same axis with aseparation in between, the sensor may be firmly affixed to the structureon each hinge member, extending through the separation (FIG. 20). Inanother embodiment employing a single hinge structure (i.e., without aseparation between two or more hinges, FIG. 29) the sensor may beaffixed at both ends on opposite sides of the hinge with a loop inbetween or other slack, such that the sensor can extend over the hingeduring flexure without stretching and tearing. At no time need thesensor be taut and the sensor is configured so that it should not becompletely extended. Where one wishes to simultaneously measure flexureand tension (FIGS. 30A, 30B), the ends of the sensor may be affixed tothe tension producing element, where the differential signal from thetwo sensing elements represents the angle of the sensor and the commonmode signal from the two sensing elements is representative of thetension applied to the sensor. The sensor may also be used to measurethe position of the pulley belt by affixing the sensor to the pulleybelt near the region where the belt is in contact with the pulley (FIG.31). As the pulley turns and the belt is bent around the pulley, thesensor will also be bent producing a signal indicative of the pulleybelt position. One may also use the subject sensors in conventional waysby adhering the sensor to a specimen, which is to be bent e.g. acantilever beam (FIG. 32). By use of the subject sensor, the adherenceof the sensing assembly to a flexible specimen is not as critical aswhen mounting a strain gage directly to the specimen, as isconventionally done, since it is only important that the sensingassembly remain in juxtaposition to the flexing specimen surface. When acoiled sensor structure is used (FIG. 33), the sensor may be used tomeasure axial rotation and changes in angles greater than 360 degrees.

A data utilization device is employed with the sensor(s). The datautilization device selected varies depending on the application of thesensors, the number of sensors involved and the information outputdesired. As described above, the strain-sensing elements comprisehalf-bridges in a Wheatstone bridge amplifying circuit (FIG. 5). Theoutput from multiple half-bridges may be multiplexed and compared to areference half-bridge voltage by a differential instrumentationamplifier. The analog output voltage may be processed by analog signalprocessing circuitry and used by an analog data utilization device. Forexample, the output voltage from a joint bend sensor may be amplifiedand filtered and used to control the velocity and/or position of a jointservo motor controlling a slave device (FIG. 15A).

Alternately, the output voltage may be digitized by an analog-to-digitalconverter and input to a microprocessor or other digital datautilization device for further processing (FIG. 15B). The digitizedsensor value may be used in various monitoring and control systems. Forexample, joint flexure may be recorded both before and after anoperation to monitor the rate and level of recovery of the joint orlimb. The digitized sensor value may also be used in a digital feedbacksystem to control the angle of an arcuate surface.

Where a plurality of sensors are involved, and the relative relationshipof the sensors is determined, as in a glove 348 used for thedetermination of the position of a plurality of joints, a computer,which is preferably portable 340 and carried by the user (FIGS. 1, 2,34A-34C), receives the digitized glove sensor data from theanalog-to-digital converter and transforms the information into symbolicoutput, e.g., letters and control signals based on a finger gesturerecognition algorithm, which can, for example, recognize fingerspelledletters. For nonvocal individuals, the glove and recognition algorithmmay be used with a voice synthesizer and other various peripheraldevices to produce a "speaking" communication device (FIGS. 1, 2,34A-34C). As each spelled letter is recognized, it may be displayed fora sighted glove-user on a visual text display 342 (e.g., an LCD screenworn as a wristwatch) or for a deaf-blind glove-user on a portablebraille display 343. Spelled words are then communicated to anotherperson by a speaker 341 or visual display 344 or sent to anotherglove-user's visual or braille feedback displays.

Other inputs to the computer system may be provided for communicatingwith the glove user 353. In the case of a hearing impaired user, ahearing person 352 can reply by typing on a miniature keyboard 345 or byspeaking through a microphone 346 into a voice recognition system. Asbefore, if the hearing impaired user is sighted, the response from thehearing person may be displayed on a visual text display 342, and if thehearing impaired person is also visually impaired, the response may bedisplayed on a braille feedback device 343.

To help prevent random hand motions from being incorrectly recognized asletters, the glove may include an orientation and/or position sensor 349(such as a mercury tilt switch or electromagnetic orientation andposition sensor) to disable the recognition when the hand is not in adesired orientation or position. The system and/or glove may also havean on/off switch 350 to disable the recognition routine or various otherfunctions. The system and/or glove may have additional function controlswitches 351.

The various input/output devices can be hard-wired to the computer, butthe devices are preferably interconnected with the computer by aninfrared link 347 or other non-wired transmission means.

In addition to a braille display in the case of a deaf-blind individual,other various tactile feedback devices may be employed such asvibrotactile displays (e.g., the Opticon™ piezoelectric tactilestimulator), electrocutaneous stimulators and robotic hands (e.g., amechanical fingerspelling hand, U.S. Pat. No. 4,074,444). The robotichand may be used in direct telerobot mode, i.e., a joint sensor value ofthe glove is processed and used to control a corresponding joint of therobotic hand (FIG. 16A), or a hand pose may be recognized using theglove with the finger-gesture recognition algorithm (i.e., the adaptivepattern recognition algorithm) and this hand pose mapped to someappropriate robotic hand configuration (which may be exactly the same asthe glove user's hand pose or different being defined by theconfiguration of the hand pose).

The communication device configuration described above may be used toaid multiple individuals to communicate over the telephone. When theglove is used in direct telerobotic mode (FIG. 16A) a deaf-blind person"reading" the robotic hand is able to sense mood and personality of theglove-user to whom he/she is communicating since the robotic hand iscapable of mimicking subtle finger motions. Similarly, a sighted deafperson could view the robotic hand output or a graphic hand model (FIG.16B) being controlled by the glove-user to whom he/she is talking. Usingthe glove to control a graphic hand for telephone communication has anadvantage over simply sending a video image of the real hand in that dueto the present bandwidth limitations of telephone lines (i.e., 300-3000Hz) a full graphic image cannot be sent and updated real-time. The imageof the hand must be data-compressed, sent, then uncompressed andredisplayed on the other end of the phone line. The uncompressed imageis normally not of good visual quality. However, using a glove, only theimportant joint motion information is sensed and sent across thetelephone line. This joint information is used to quickly construct agraphic hand model for viewing.

An earlier embodiment which was found to be satisfactory is described inthe following figures. Referring now specifically to the drawings, FIG.1 is a functional block diagram of a communication system in accordancewith the invention and FIGS. 2A and 2B illustrate a hearing person and adeaf-blind person communicating using the system. The system providesexpressive and receptive communication capabilities for deaf, deaf-blindand nonvocal persons. Although the preferred embodiment of the system isportable, as shown in FIG. 1, some components of the present prototype(FIG. 7) are too large to be carried.

The prototype system consists of a Motorola 68HC11 microcomputer board,with various interchangeable peripheral devices that encode and displayinformation. Each user can select the peripheral devices that bestsatisfy his individual communication needs. In the Figures, the arrowsdenote the directions of information flow.

As shown in FIG. 2B, a non-vocal deaf-blind user is transmittingsynthesized speech to a hearing person. The non-vocal personfingerspells while wearing a glove equipped with uniquely designedstrain gage flex sensors. The microcomputer receives finger joint angleinformation from the instrumented glove 12 (FIG. 1, upper left). Themicrocomputer executes a letter recognition algorithm based on the jointangle information.

The recognition algorithm selects the most probable letter from an arrayof pre-stored hand formations that characterize the user'sfingerspelling "signature." Once the letters of a word have beenrecognized and stored, a read-out command is given and the entire wordis spoken to the hearing person (FIG. 2A) via a DECtalk speechsynthesizer. The nonvocal individual wears a small output speaker (FIG.1, upper right) as a "speech pendant" under his shirt. In effect, thenon-vocal person's information output device is a "talking glove." Aftereach word is spoken, the recognition algorithm adapts itself to reducesensitivity to ongoing variations in letter formation and sensorplacement.

A hearing person replies to the deaf user by typing on the keyboard of aRadio Shack pocket computer (FIG. 11, lower right and FIG. 2A). Thekeyboard has a wireless link (infra-red) to the microcomputer so it canbe carried in the shirt pocket of the deaf or deaf-blind individual andhanded to the hearing person. The small keyboard has a liquid crystaldisplay (LCD) that states the deaf person's name and requests messageinput from the hearing individual. Data entered on this keyboard istransmitted to the microcomputer and displayed for the deaf person usingone of the methods described below. It is possible for several keyboardsand microcomputers to share a common communication channel so that morethan one hearing individual could participate in the same conversationwith more than one deaf or deaf-blind person.

An IntroVoice voice recognition system can be integrated for use incertain structured environments, such as at home or at work. Individualstrained on this system can speak directly to the deaf individual. Theirspeech is detected by a small, directional microphone (incorporated intothe deaf person's speech pendant) and converted into text by the voicerecognition equipment (FIG. 1, lower right)

Text generated by the hearing person, using either the pocket keyboardor voice recognition system, is displayed for the deaf individual in avariety of ways (FIG. 1, lower left). For sighted deaf persons, theentered message is displayed on a small Seiko LCD monitor work like awristwatch. For deaf persons with reduced vision, text can be output ona high-visibility large-character LED display. For a deaf-blind user,information is presented on a specially designed mechanical brailledisplay (FIG. 2B). The braille display can fasten to the belt for usewhile walking and standing but detach for desk-top use. Braille displayscan also include a calculator mode, and a data-bank function to recordnames, phone numbers and appointments.

FIG. 3 is a plan view of a glove 20 in accordance with one embodiment ofthe invention. The glove has fourteen strain gage sensors inserted intopockets 22 sewn in the glove, to sense joint flexure of the fingerswhile fingerspelling letters. FIG. 4 is a side view of a finger and astrain gage sensor for sensing the flexure of the finger joint. Inaccordance with one embodiment, two strain gage resistors are mounted ona 1 mil polyimide backing material so that during flexure, one straingage is in tension while the other strain gage is in compression. A 120ohm high elongation annealed Constantan resistor strain gage fromMicroengineering II (model PAHE-DF-750-120 LEN) has been used.

Data from the joint sensors is used by an adaptive pattern recognitionalgorithm to identify the intended letter from the fingerspelling handformation. Recognized letters and phrases can also be sent to otherdevices such as a TDD or computer. In effect, the user's informationoutput device is a "talking glove."

The instrumented glove is attractive and comfortable. The fingertips andpalm of the glove preferably are removed to increase functionality andventilation. Exposed fingertips permit the user to reliably grasp suchobjects as a pen or coffee cup, while the open palm allows a deaf-blindperson to accept manual fingerspelling. Material covering the middleknuckle on the palm side of the glove can also be removed to preventbinding. As noted above, each angle sensor consists of two long flexiblestrain gages mounted back to back. The strain gage assembly resides in aguiding pocket sewn over each joint. The maximum strain seen by eachgage is adjusted by varying the thickness and elastic modulus of theplastic backing to which the gage is mounted. The backing thickness isselected to maximize the output signal without significantly reducingthe fatigue life of the gage.

The gages are wired in a Wheatstone bridge configuration that employs ananalog multiplexer to select which bridge voltage is to be sampled by ananalog-to-digital converter, as illustrated in FIG. 5.

Each sensor has a pair of strain gages, one in tension (T) and one incompression (C), which are used with fixed resistors (R) in a Wheatstonebridge arrangement. Each sensor bridge output is multiplexed through MUX30 (AD7506, for example) to amplifier 32 (AD624, for example). Thevoltage offset of each input to MUX 30 can be varied by an offsetpotentiometer (not shown). The output of amplifier 32 is then applied toA/D converter 34, and the digital output from converter 34 is appliedthrough bus 36 to the computer. The voltage range of the input to A/Dconverter 34 can be selected by choosing the appropriate gain ofamplifier 32.

The core of the control software is an adaptive pattern recognitionalgorithm. In the algorithm, each sensor value is treated as onecomponent of an overall "hand state vector," i.e., each joint anglerepresents one axis in an n-dimensional "hand space" where n is thenumber of instrumented joints (typically n=14). As the hand moves, thehand state vector traces a trajectory through hand space. This is shownfor a three-dimensional space in FIG. 6.

Initially, sensor information must be stored for each letter to informthe system of the user's fingerspelling "signature." The sets of storedsensor values are represented figuratively by "letter beacons" in handspace (FIG. 6). When the algorithm detects a reduced hand statevelocity, a letter formation is suspected, and the nearest beacon isdetermined. If the instantaneous hand state lies within the closestbeacon's "recognition ball," denoted figuratively by the dashed innerspheres in FIG. 6, the corresponding letter is recognized and placed ina work-assembly buffer. To avoid unintentional repeated recognition ofthe same character, hysteresis is introduced into the letteridentification algorithm by requiring that the hand state vector leavethe outer sphere before a new character is determined. Backspacing ispermitted, and a "say it" beacon exists which, when recognized, causesthe letters in the work-assembly buffer to be spoken by a DECtalk voicesynthesizer. When a word is spoken, the letter beacons are adaptivelyupdated to reflect their new, most probable positions in hand space.Continuous adaptation reduces sensitivity to ongoing variations inletter formation and sensor placement.

FIG. 7 is a functional block diagram of a prototype communication systemdeveloped at Stanford University in which the recognition algorithm isimplemented in an IBM AT personal computer. A Motorola 68HC11microcomputer communicates with the IBM AT and also controls theoperation of the peripheral devices.

FIGS. 8A and 8B are a flow diagram of an earlier hand pose recognitionalgorithm which was found to be satisfactory. The recognition algorithmwas implemented in software, runs on the IBM AT computer and recognizesfingerspelled letters. Additional recognition algorithms, andmodifications to the earlier recognition algorithm, which provideimproved recognition performance are described later.

FIG. 9 is a flow diagram of the Wait to Exit recognition ball routineand FIG. 10 is a flow diagram of the Check for Inputs routine as used inthe letter recognition algorithm of FIGS. 8A and 8B. FIG. 11 is a flowdiagram of the I/O and sampling software running on the Motorolamicrocomputer.

A printout of the software illustrated in FIGS. 8A, 8B, 9 and 10,written in Turbo Pascal, is attached hereto and incorporated byreference. A printout of the software illustrated in FIG. 11, written inForth, is also attached hereto and incorporated by reference.

Referring now to FIG. 8A, the flow diagram for the letter recognitionalgorithm software begins with the initialize/alter default recognitionparameters at 50. These parameters include the following:

1. recognition/unrecognition ball radii;

2. beacon-beacon hysteresis margin that permits unrecognition if"substantially closer" to a new beacon;

3. recognition confidence margin, required to recognize one beacon overthe next nearest neighbor;

4. recognition sampling period;

5. maximum velocity that can denote a pause;

6. number of sample points required to abide by parameters (1), (3) and(5), supra;

7. joint weighting;

8. additional letter `c` recognition pause--to prevent incorrectrecognition;

9. voice selection (e.g., male, female, child);

10. select Dexter (mechanical fingerspelling hand) on/off, (FIG. 16A);and

11. user name selection.

Next, the user's hand formation template, to be used during recognition,is loaded at 52. If the user is new to the system, then a defaulttemplate is loaded. The hand formation template can be altered at 54 ifdesired by the user forming each of the letters and symbols sequentiallyand recording the sensor data corresponding to each letter beaconlocation.

Next, all variables are reset at 56 and then the Fetch Gesture routineis activated.

Determination of a pause at a beacon is made at 58. For the Pause at aBeacon routine at 58 in FIG. 8A to produce a True, the following threequestions must answer true: (1) pause? (2) inside a recognition ball?and (3) closer to the closest beacon than to the second closest beaconby more than the recognition confidence margin?

If Paused, then the Wait to Exit recognition ball routine (FIG. 9) isactivated. If not paused, then the Check for Inputs routine (FIG. 10) isimplemented. Next, previous sensor data values are shifted back in timeone sample period prior to reading current sensor data to be used torecognize a new beacon for readout and storage. New glove data is readand the closest beacon is then determined using a "Least Squares" (L.S.)mathematical determination in 62. The Find Closest Beacon routine 62,has been optimized to very quickly search through all the possiblebeacons and record the closest and second closest beacons. The quicksearch is achieved by first calculating the least squares distance tothe previous closest and second closest beacons. The remaining letterbeacon distances are calculated in order of decreasing probability asdetermined from letter probabilities of the English language. Since theleast squares method involves summing up the squares at the deviationsof the selected beacon's joint sensor value, the calculation is checkedafter each squared deviation is added to the running total. If theintermediate sum is already greater than the previous second closestbeacon, the L.S. distance calculation for that beacon is terminated andthe next beacon's distance determination is begun. In many cases, theprevious closest and second closest beacons will again be the newclosest and second closest beacons, so after the L.S. distances to theseprevious closest beacons has been calculated, only the first term in theL.S. sum of the remaining beacon distances will need to be calculated.If the new closest beacons are not the previous closest beacons, thesearch algorithm calculates the beacon distances in order of decreasingprobability so that as many of the following distance calculations aspossible can be terminated prematurely. Thus computation time isdramatically reduced.

If it is determined from the new glove data that the user has not pausedat a beacon, then a recognition delay is performed and new glove data isagain read using the cycle described above. If it is determined from thenew glove data that the user has paused at a beacon, then the lettercorresponding to the beacon is displayed on the IBM monitor and thensent to the microcomputer for processing and display on the variousperipheral devices shown in FIG. 7. Thereafter, the flow diagramproceeds to FIG. 8B where an escape variable is tested. If the escapevariable tests positive, then the recognition loop is exited.Thereafter, the user can choose to alter recognition parameters, returnto the recognition loop or exit the program. If it is desired to exitthe program, the user's letter beacon template is stored to hard disk tobe retrieved later at 52 when the program is restarted.

FIG. 9 is a flow diagram for the Wait to Exit recognition ball routine.After checking for inputs and reading the new glove data to determine anew hand state position, the closest beacon is determined by using thenumerical method of Least Squares. If the new hand state is still"closest" to the previously recognized beacon, then it is determinedwhether the unrecognition ball of the previous beacon has been exited.If not exited, the Wait to Exit recognition ball routine is startedagain. If it is determined that the unrecognition ball of the previousbeacon has been exited, the Wait to Exit recognition ball routine isexited, returning control to the main program at 60 (FIG. 8A). If thenew hand state is not closest to the previous beacon, then a calculationis made of the distance to the previous closest beacon. If the new handstate has exited the previous beacon's unrecognition ball then the Waitto Exit recognition ball routine is exited and a new beacon can berecognized. However, if the new hand state has not exited theunrecognition ball of the previous beacon, then it is determined whetherthe new hand state is closer to a new letter beacon by more than thebeacon-to-beacon hysteresis margin. If yes, the Wait to Exit recognitionball routine is exited. If no, the Wait to Exit routine is repeated.

FIG. 10 is a flow diagram of the Check for Inputs routine. Basically theroutine is looking for an input received from the microprocessor, akeypad message, voice recognition system or IBM keyboard input. If suchmessages are present, then they are displayed.

FIG. 11 is a flow diagram of the peripheral control and sensor samplingroutine running on the Motorola microcomputer. Basically, the routinecommunicates with the IBM computer and controls the operation ofperipheral devices shown in FIG. 7.

Although the recognition algorithms and related ideas provided below aredescribed specifically in the context of recognizing finger gestures theconcepts are general and may be used in various applications includingapplications where it is desired to recognize a vector of features thatare continuously varying in time. The newly developed tree-structuredneural net classifier, provided as a classification option in therecognition algorithm, may be used in many general classificationapplications requiring quicker training and classification than standardfeed-forward artificial neural networks.

In finger-gesture (hand pose) recognition mode each sensor value istreated as one component of an overall "hand state vector," i.e., eachjoint angle represents one axis in an n-dimensional "hand space" where nis the number of sensors, e.g. instrumented joints. The number ofsensors is typically at least about 14, preferably at least about 16,and may be 24 or more. As the fingers move, the hand state vector tracesa trajectory through hand space, shown for a 3-dimensional space in FIG.6. Initially, a training set is created where sensor information isstored for each hand pose to inform the system of the user'sfinger-gesture statistics, i.e., "signature." Representative points,e.g., hand pose mean values, for the training set of stored sensorvalues are represented figuratively by "beacons" in hand space (FIGS. 6,36).

The overall recognition algorithm includes a gesture parsing routine anda classification routine (FIG. 37) which work together to extract thedesired hand pose from continuously varying finger and wrist motion.When the parsing routine, which monitors the joint dynamics, detects ahand state velocity minimum point (FIG. 30), one of variousclassification algorithms is used to determine the "nearest" predefinedhand pose (beacon) corresponding to the hand state sample. The subjectinvention includes an option to select either a statistics-basedclassifier, or one of two artificial neural network classifiers. If theselected classifier determines that the instantaneous hand state lies"close enough" to a predefined hand formation (where closeness ismeasured by some appropriate distance metric) the corresponding handpose is recognized. To avoid unintentional repeated recognition of thesame hand formation, hysteresis 360 is introduced into thefinger-gesture recognition algorithm by requiring that either the handstate vector get "far enough" away from the recognized hand formation orthe hand state velocity to appreciably increase, before a new hand poseis determined. At select times (for example after a fingerspelled wordis spoken), the hand pose beacons may be adaptively updated to reflecttheir new, most probable positions in hand space. Continual adaptationreduces sensitivity to ongoing variations in hand pose formation andsensor placement.

A simple adaptation scheme is a filter which only uses the most recent nsamples of a letter to determine the hand pose beacon mean and otherstatistics. So, when a hand pose is recognized, the sensor data from thehand pose recognized n samples ago is replaced with the new sensor data.

When the recognition algorithm is used in the communication system aspreviously described, the recognized hand poses represent letters of thefingerspelling alphabet. When a hand pose is recognized itscorresponding letter label is placed in a word-assembly buffer.Backspacing is permitted and a "say it" hand pose (i.e., beacon) alsoexists. If a backspace hand pose is recognized, the erroneous hand poseis not adapted. Once the "say it" hand formation has been recognized,the entire word in the word-assembly buffer may be output as synthesizedspeech to the hearing person preferably via a small speaker worn by thedeaf individual (FIGS. 1, 2, 34A-34C).

As provided above, the subject invention includes an option to selecteither a statistics-based classifier, or one of two artificial neuralnetwork classifiers. The selected classifier utilizes the storedtraining (hand state) vectors to classify a sample vector taken atrecognition time. When the statistical classifier is used, each sensorvalue is modeled as a normally distributed random variable with meanequal to the centroid of the hand pose samples, and thus, the hand statevector is actually a multivariate normal random vector. The multivariatenormal model has been verified by the inventors through Q--Q plots andstatistical hypothesis testing. The statistical classifier is based onBayesian Decision Theory which uses a priori probabilities about eachhand pose as well as a posteriori probabilities to select the hand posethat is least probable to be in error. The a posteriori probabilitiesfor each hand pose are calculated for a given sample vector usingknowledge of the probability density function (pdf) obtained from thestored training set.

Using the statistical (Bayesian) classifier and the assumption that thesensors values are random variables of a multivariate normaldistribution, we define the square of a generalized distance measure,d_(i), to be

    d.sub.i.sup.2 =r.sub.i.sup.2 +In|S.sub.i |-(2×In P(Θ.sub.i)!),

where r_(i) ² =(x-m_(i))^(t) S_(i) ⁻¹ (x-m_(i)) and is called theweighted sum squared error (WSSE) or squared Mahalanobis distance forhand pose i, x is the random hand state vector, m_(i) is thevector-valued mean of the samples for hand pose i, S_(i) is the samplecovariance matrix of the random hand state vectors for hand pose i andP(Θ_(i)) is the a priori probability of hand pose i being the nextintended hand pose. Note that the equation for r_(i) ² produces a familyof concentric isovalue hyperellipsoids in x for each hand pose, onecurve for each value of r. Similarly, the equation for the generalizeddistance squared (GDS), d_(i) ² produces a family of isovaluehyperellipsoids for each hand pose. FIG. 36 shows two concentrichyperellipsoids for each of two letter hand poses. Therefore, theclassification rule reduces to selecting the symbol corresponding to thefamily of hyperellipsoids in which the hand state sample lies inside thehyperellipsoid of smallest isovalue. In FIG. 36, point p₃ is closer to"A." Mathematically, the classification rule requires calculation of theWSSE distance from the hand state sample to the centroid (mean) of eachhand pose, adds two terms which may depend on i, and then selects thehand pose which is "closest." The WSSE distance calculation is aquadratic form, which in case of the Bayesian classifier, uses theinverse of the sample covariance matrix, S_(i), as a weighting matrix,W. As a result, the directions in hand space, as determined by theeigenvalues and eigenvectors of S_(i), that had the most variabilityduring training are given the least weight during recognition. Note thatfor the special case where the weighting matrix for a given hand pose isset to the identity matrix, I, then all random variable components forthat hand pose are considered to have the same variance. Thus, allcomponent errors are weighted equally and the WSSE distance measurereduces to simply a sum squared error (SSE) distance measure, and thehyperellipsoids reduce to hyperspheres. (Note: The sum squared error(SSE) distance is sometimes referred to as the least square (LS)distance, although, more accurately, "least squares" is a procedure forcalculating regression coefficients subject to the constraint that thesum of the squares of the regression errors is minimized.)

The generalized distance from the sample hand state to the expectedclosest hand pose is calculated first to reduce the expected number ofterms that are necessary to be calculated in the generalized distance toany of the remaining hand poses, before that particular remaining handpose can be eliminated as a "closest" candidate. For example, if theclosest hand pose were known ahead of time and the distance to itcalculated first, then the distances to each of the other candidate handposes is further. Therefore, since the WSSE distance calculation of thequadratic form y^(t) Wy consists of a sum over i and j of terms w_(ij)y_(i) y_(j), the summation calculation is preferably stopped when therunning sum exceeds a critical distance, for example the distance to theclosest or second closest hand pose, or some predefined maximumacceptable distance. To further increase the chances that the runningdistance sum will pass the critical distance and be terminated as soonas possible, the weight matrix is spectrally factored into a summationof n terms, i.e., W=S⁻¹ =Σ_(i) (1/λ_(i))v_(i) v_(i) ^(t), for i=1 to n(the number of sensors), where λ_(i) is the I^(th) eigenvalue and v_(i)is the i^(th) eigenvector, or principal axis in hand space, of thesample covariance matrix. (Note: Since eigenvectors are also called theprincipal axes, this method of spectrally factoring into eigenvectors isalso referred to as Principal Component Analysis.) Then the generalizeddistance calculation becomes

    d.sub.i.sup.2 =Σ.sub.i (1/λ.sub.i)(x.sup.t v.sub.i).sup.2 +k.sub.1i +k.sub.2i,

where k_(1i) =In|S_(i) |, k_(2i) =-2×In P(Θ_(i))!, and i=1 to n. Notethat without loss of generality, m, was set to zero in the equationabove for clarity which simply means that we are measuring the distanceto a hand pose with the coordinate system placed at the center of thehyperellipsoids corresponding to that hand pose. Since x^(t) v_(i) isthe projection of x onto v_(i), we can see, in the two sensor example ofFIG. 36 that x^(t) S⁻¹ x>c², where c is our critical isovalue, just byexamining the first term of the sum, i.e., |λ₁ ^(-1/2) x^(t) v₁ |>|c|.By calculating the WLS distance in this spectrally factored form we havealso eliminated the need to invert the matrix, S, that becomesill-conditioned as one of its eigenvalues approaches zero (and thus |S|goes to 0), implying that one of the directions in hand space is notbeing used.

In addition to finding the hand pose beacon which is closest to thesample hand state when the parse routine denotes a pause, other criteriaare preferably imposed to help the recognition routine perform morereliably. For example, once the distance to the closest beacon iscalculated the distance is compared to a "recognition maximum" distancefor that beacon. If the distance from the current hand state to theclosest beacon is greater than the beacon's recognition maximum distancethen recognition is not permitted. This insures that the closest beaconis "close enough" to be considered. For the feed-forward neural netclassifier which uses sigmoidal output units and has outputs rangingfrom -1 to +1, the recognition threshold value usually ranges from -0.3to 0.99 and depends on the application. For a fingerspellingapplication, the threshold value is preferably set to 0.2. Note that forthe typical feed-forward neural net classifier, a sigmoid output nearerto +1 denotes that the sample hand state is nearer to the hand posecentroid and so the threshold is actually a minimum value, below whichno hand pose is recognized.

For the statistical classifier the generalized distance to the closestbeacon must lie inside a recognition hyperellipsoid 351 of predeterminedisovalue (FIG. 36). Since individual sensor values are modeled asnormally distributed random variables and since the WSSE distancesquared to any beacon is the sum of squares of normal random variables,then the square of the WSSE distance measure has a Chi-Squareddistribution. Therefore, it is preferred to specify the recognitionhyperellipsoid such that if the hand state sample is outside thathyperellipsoid then the probability of error in not recognizing thecorresponding hand pose is below a specified maximum, α. Mathematicallyspeaking, this is equivalent to

    Pr{d.sup.2 <X.sub.n.sup.2 (α)}=1-α

where X_(n) ² (α) is the value for an Chi-Squared random variable with ndegrees of freedom, and α is the specified maximum probability of error.

The recognition algorithm may require a certain level of confidencebefore selecting one of two fairly close beacons. For example, it ispreferred that a recognition confidence margin be used such that, evenif all other recognition requirements are satisfied, the closest beaconmust still be closer than the second closest beacon to the sample handstate by more than the recognition confidence margin.

Hysteresis 360 (i.e., a dead-band) may be added to the recognitionalgorithm by requiring the hand state vector to travel further from therecognized beacon before the beacon can be recognized again (FIG. 36).For the neural net routine, this corresponds to requiring the sigmoidaloutput to decrease below an "unrecognition" threshold value (below therecognition threshold) before the same letter can be re-recognized. Forthe statistical routine, the unrecognition threshold becomes an "outer"hyperellipsoid 362 (FIG. 36) beyond which the hand state must passbefore re-entry into the recognition hyperellipsoid will re-recognizethe hand pose.

In some cases two beacons can be so close that one is actually insidethe other's recognition region. What is more often the case is that thesample hand state lies inside more that one recognition region at thesame time (e.g., p₃, FIG. 36). The problem that results is that thesample hand state may be much closer to one beacon, but because adifferent beacon was recognized first, and the hand state still hasn'tleft its unrecognition region, the closer beacon isn't recognized. Toremedy this situation, an unrecognition confidence margin is set suchthat if the hand state is closer to a new beacon than the currentlyrecognized beacon by more than the unrecognition confidence margin, thenthe currently recognized beacon is unrecognized and the new beacon isrecognized.

As mentioned above an artificial neural network structure may be used toclassify sample hand states. The subject invention provides two possibleneural net structures which may be used to train hand poses andrecognize sample hand states. The first is a feed-forward neural net asshown in FIG. 39A. At the far left are the input nodes. There ispreferably at least one input node per sensor. In the simplest casethere is one input node for each sensor value. In more complicated casesthere may be position, velocity, acceleration, position squared and thelike, inputs for each position sensor. Input values are scaled to occurin the range of -1 to +1. Each input value is connected to one or moreof the nodes in the middle layer. The nodes in this layer are calledhidden units. Each unit sums its weighted inputs (including a weightedconstant input, called the bias) and then passes the sum though asigmoidal squashing function (FIG. 39B) that produces an output between-1 and +1. Each of the outputs of the hidden units may be used as inputto additional hidden layers or may be input to one or more units of theoutput layer. As with the hidden units, the output units sum weightedoutputs from the previous layer and then pass the sum though anadditional function which may be sigmoidal, linear or other appropriatefunction.

There are various ways to train the network to classify hand statevectors. The preferred training method is to assign each hand pose to berecognized to a different output unit. When training the net torecognize a vector known to come from class i, a desired output vectorconsisting of -1 at the output of all output units except an output of+1 at unit assigned to class i, is used. The Back-propagation (SigmoidLMS) learning algorithm is conveniently used to train the weightsinterconnecting the various layers of units to produce an output vectorwhich is close to the desired output vector for each input vector in thetraining set. Learning rates usually in the range of 0.001 to 0.5 areused, whereas a rate of 0.1 is preferably used to train hand posescorresponding to letters of the fingerspelling alphabet. Techniques suchas momentum, simulated annealing and the like may be used to enhance thelearning algorithm.

When the training method described above is used along with appropriateoffset, scaling and normalization to produce outputs in the range from 0to +1 with all outputs summing to 1.0, the output value from aparticular unit in the output layer is representative of the probabilitythat the input sample hand state should be assigned to the classcorresponding to that output unit. When classifying a sample hand stateinput vector, the class corresponding to the output unit with thelargest output, i.e., highest probability, is selected as "closest."However, as explained above, when the recognition algorithm of thesubject invention is used, other criteria may also be required beforethe closest beacon will be recognized.

The subject invention also provides a newly developed neural networkalgorithm that may be used to perform general classification tasks, andspecifically, to classify hand poses. The algorithm is a tree-structuredneural network classifier (TSNNC). During training of a labeled dataset, a decision tree is constructed which is then used during thesubsequent recognition phase to classify a sample input vector based ona number of decisions, one decision per node of the tree. FIGS. 40A and40B show two example sets of clusters of labeled training data pointsand the corresponding possible resulting binary trees produced by theTSNNC. This TSNNC algorithm will always produce a zero-error solution onconvex training sets. The maximum rate (i.e., length of the tree) isguaranteed to be less than the number of data clusters to be classified.Therefore, the number of multiply/adds less than a simple feed-forwardneural net with the same number of inputs, output units and with nohidden units.

FIG. 41 provides a flow diagram of the TSNNC algorithm for constructinga binary decision tree. The first step to construct the decision treefor a given labeled data set is to initialize a hyperplane to passthough the centroid of all the samples. Note that a single sigmoidalunit (FIG. 39B) is a function which passes the sum of its weightedinputs through a sigmoidal squashing function and returns a real valuedresult between -1 and +1. When the equation for the sum of the weightedinputs is plotted in weight-space, i.e., each axis represents the valueof one of the d input weights, the resulting curve is called ad-dimensional hyperplane. The hyperplane is initialized parallel to theeigenvector corresponding to the minimum eigenvalue of the scattermatrix of the points to be separated. Next, using a single sigmoidalunit, all data points are classified to belong to either the newlycreated -1 class or the +1 class. By belonging to the -1 class, it ismeant that the desired sigmoidal unit output value of all pointsassigned to the -1 class is -1. Similarly, the desired unit output valueof all points assigned to the +1 class is +1.

The individual data sample classification error is defined to be thedifference between the desired sigmoidal unit output value (asdetermined by the assigned class label of either -1 or +1) and theactual unit output. For each original cluster of labeled data, the sumsquared error (SSE) is calculated that would result if all the samplesin that cluster that were assigned to the newly created -1 class were tobe assigned to the +1 class. Similarly, the SSE is calculated that wouldresult if all the samples in the cluster in question that were assignedto the newly created +1 class were assigned to the -1 class. For eachoriginal cluster, the value which would induce the least SSE is chosenas the desired class value (i.e., -1 or +1) for all the samples of thecluster.

Using an appropriate learning algorithm, such as Back-propagation(Sigmoid LMS), the unit is trained to separate all the clusters into thetwo classes. Simulated annealing may be used to help prevent thesolution from getting stuck in a local minimum. The intuition behind therelabeling and training is to attempt to get all data points in a singlecluster on the same side of the dividing hyperplane, and to choose aclass value for each cluster such that the hyperplane is required tomove the least.

If after training a unit, the final mean squared error (MSE) over thesample points of all clusters divided by that unit is "small enough,"and all clusters are at a leaf node of the tree (i.e., there is only onecluster assigned to the -1 class and one to the +1 class for aparticular unit), stop, the training algorithm is finished and adecision tree has been constructed. If the final mean squared error(MSE) from training a unit is "small enough," but not all clusters areat a leaf node of the tree, then that unit is considered "fully trained"and a new unit (hyperplane) is initialized, similar to above, whichfurther divides all the samples of the clusters assigned to that class.Now proceed in the same manner as was done previously where all samplepoints of a particular cluster are assigned to a class such that theinduced SSE is minimized. The newly added unit is then trained until itconverges to an acceptable MSE. If again, all clusters are not leafnodes, a new unit (hyperplane) is initialized and trained. In otherwords, to continue, the samples assigned to a single -1 or +1 class arefurther subdivided with more hyperplanes until the last hyperplanedivides only two clusters, which are at leaf nodes of the tree.

The procedure just described works fine for designing a tree to classifyclusters that are linearly separable at each branch. However, when theclusters at a branch are not linearly separable the learning algorithmwill not converge to a sufficiently small MSE. If the clusters to beseparated by a unit are not linearly separable but are convex, thefollowing modification is made to the algorithm. The cluster, call it a,that is contributing most to the total MSE is selected, all its samplesare ignored and the unit is further trained on the samples from theremaining clusters. If the training converges this time, that unit isconsidered fully trained. However, with this unit, cluster a has justbeen divided into two pieces, α₁ and α₂, where α₁ is a sub-clusterassigned to the -1 class and α₂ is a subcluster assigned to the +1class. The TSNNC algorithm continues from this point as if it has justdivided the clusters without splitting a cluster into two. If afterremoving the samples from cluster α, the training routine still does notconverge, the procedure of finding the cluster which now contributesmost to the MSE is repeated, all its samples are ignored and training iscontinued. The process of ignoring clusters of samples will occur atmost m-2 times per unit before the training for that unit will converge,as long as the clusters are convex, and where m is the number ofclusters to be separated by that unit. In the worst possible case whereeach unit must ignore m-2 clusters, or where the unit is otherwise ableto separate only one cluster from the m-1 remaining clusters at a time,leaving m-1 clusters to be separated by the next unit, the maximumlength tree of length m-1 is constructed. This means that it willrequire m-1 binary decisions to classify an arbitrary sample vector.

The recognition process for the constructed decision tree is very quickand simple. Whereas sigmoidal units were required by the learningalgorithms suggested, simple binary signum units are used duringrecognition. That is, at each branch in the tree, the inputs areweighted by the weights as determined in the training procedure, summedand the sum simply compared to zero. If the sum is greater than zero onechild branch is followed, if the sum is less than zero the other childbranch is followed. Once a leaf node is reached the sample input testvector is classified as belonging to the same class as the clustertrained to correspond to that node.

The main drawback of the TSNNC algorithm is that it is harder to tellhow "close" the test vector was to the centroid of any cluster so suchideas as (un)recognition thresholds and (un)recognition confidencemargins and the like may be harder to employ. One solution to thisproblem is to simply use the TSNNC to quickly determine the closestbeacon and then go back and calculate the generalized distance or neuralnet distance from the test vector to that same beacon, and use thatdistance when comparing to (un)recognition thresholds, etc.

As previously mentioned, a gesture parsing routine works closely withthe chosen classification routine to select the desired hand pose at theappropriate time. The parsing routine calls on several other routinesand functions to help it make a parsing decision. For example, theparsing routine communicates with a pause detector which, among otherduties, determines when the hand state velocity reaches a minimum point(FIGS. 38, 42). The parsing routine also takes the distances from thehand state vector to the various closest hand poses and compares thesedistances to the (un)recognition thresholds and (un)recognitionconfidence margins.

To better understand the pause detection routine, refer to FIG. 38 whichshows an example hand state velocity vector magnitude (i.e, ∥v∥) versustime curve with various points of interest marked. As is seen, not allpoints at a velocity minimum are considered "pause points." There areseveral requirements for a velocity minimum to be considered a pausepoint. The first requirement is that prior to the velocity minimum thehand state must have been considered "not paused." This requirementreduces the probability that a hand pose will be incorrectlyre-recognized. If the hand state is not considered to be paused and thehand state velocity dips below a lower velocity threshold (V_(lower))and reaches a minimum, the hand state is considered paused. If the handstate is not paused and the hand state velocity is not below V_(lower),there are two ways for a pause to be detected. A pause will be declaredat a velocity minimum point which occurs after the velocity has beendetermined to fall a predefined percentage below the maximum velocity(v_(max)) achieved since the hand state was determined not to be paused.Mathematically, this is saying that there exists a velocity pausethreshold, T_(vp) (which may depend on v_(max)), such that if (Δv_(max)/v_(max))<T_(vp) <0 then a pause will be declared at the next velocityminimum. Note that the quantities are all less than zero since velocityis decreasing and thus Δv_(max) is negative. A pause will also bedeclared at the first velocity minimum if, once a set amount of time haspassed since a velocity maximum, the average deceleration is greaterthan a preset amount. This is equivalent to saying that there exists anacceleration pause threshold, T_(ap) (which may depend on v_(max)), suchthat if (Δv_(max) /Δt₁)<T_(ap) <0 (where Δt₁ is ≧t_(1min)) then a pausewill be declared at the next velocity minimum. In summary, among othercriteria for a parse to be determined and the closest hand pose to berecognized, the hand state must be paused, the hand state must be "closeenough" to the closest hand pose, the hand state must be sufficientlycloser to the closest hand pose than the second closest hand pose, andthe no hand pose may be currently recognized.

Once the hand state is considered paused, there are several ways for itto be "unpaused." If the hand state was declared paused at a velocityminimum that was below V_(lower), then the hand state is declaredunpaused when the velocity surpasses an upper velocity threshold,V_(upper). If the hand state was declared paused at a value aboveV_(lower) then it may be declared unpaused whenever either of twocriterion are met. One way for the hand state to be declared unpaused isif the velocity rises a predefined percentage above the velocity minimumpoint. This is equivalent to saying that there exists a velocity unpausethreshold, T_(vu), (which may depend on v_(min)) such that if (Δv_(min)/v_(min))>T_(vu) >0 the hand state is declared unpaused. The hand stateis also declared unpaused if, once a set amount of time has passed sincethe velocity minimum, the average acceleration is greater than a presetamount. This is equivalent to saying that there exists an accelerationunpause threshold, T_(au) (which may depend on v_(min)), such that if(Δv_(min) /Δt₂)>T_(au) >0 (where Δt₂ ≧Δt_(2min)) then the hand state isdeclared unpaused. In summary, among other criteria, a hand pose isconsidered unrecognized whenever the hand state vector is declaredunpaused, or it is "far enough" away from the recognized hand pose or itis "close enough" to a new hand pose.

To determine the point where the velocity reaches its minimum, the valueof instantaneous acceleration (i.e., d∥v∥/dt) may be used (FIG. 43),since when the magnitude of velocity, ∥v∥, is decreasing to a minimum,the value of the hand state acceleration is increasing toward zero froma negative value. Therefore, by simply watching for when theacceleration reaches the acceleration threshold value just below zero,the velocity minimum point will be satisfactorily determined. By havingthe acceleration threshold value just below zero, this technique has theadded benefit that it will also detect when the velocity is slowlyapproaching a constant value.

Several vector norms may be used to convert the velocity andacceleration vectors to scalar values that can be compared tothresholds, etc. The preferred vector norm is the 2-norm (i.e.,Euclidean norm), however other popular norms such as the 1-norm,infinity-norm or general p-norm may be used. To calculate the 2-norm ofa vector, the components of the vector are squared, added and then thesquare root of the sum is taken. However, this calculation implies thatmotion from all sensors should have equal weighting. This is notnecessarily a good idea. Take, for example, the case of finger-gesturerecognition. Different sensors may be used which have different inherentgains built in, or similar sensors may be placed on different parts ofthe hand which move through different angular ranges. The signal fromone sensor may be known to be of less importance than the signal fromanother, if so, weighting should reflect the relative importance.

When the statistical classifier is used, the norm for the velocityvector may use the same weights that are used by the classifier, i.e,W=S_(i) ⁻¹. Mathematically, this is equivalent to ∥v∥₂,w =(v^(t)Wv)^(1/2), where the subscript 2 means the 2-norm, and the subscript Wmeans weighted by matrix, W. By using the inverse of the samplecovariance matrix to weight the hand state velocity vector, thosevelocity components that correspond to directions in hand space that hada lot of variance during training of a particular hand pose will notreceive a lot of weight when attempting to detect a pause. The sameweight matrix, i.e., W=S_(i) ⁻¹, may also be used to weight the handstate velocity vector when one of the neural net classifiers is used. Touse this weighting scheme for any of the classifiers, the closest handpose must first be determined so the appropriate weighting matrix may beselected. Therefore, for every hand state sample the closest and secondclosest hand poses and their corresponding distances are determined. Ifall other criteria for recognition pass, e.g., the closest hand pose isclose enough, and sufficiently closer than the second closest hand pose,then the weighted velocity norm and the associated acceleration arecalculated using the weighting matrix corresponding to the closest handpose, and it is determined if the hand state is paused.

An alternate weighting scheme is to weight each velocity component bythe gain necessary to expand the positional range of the correspondingsensor value to a chosen standard value. For example, if values fromsensor one only ranged between 0 and 25, while values from sensor tworanged from 0 to 50, then if the standard value were chosen to be 100,the gain for sensor one would be four, while the gain for sensor twowould be two. These same gains would be applied to the first twocomponents, respectively, of the velocity vector norm calculation.

The effectiveness that the pause detection routine has in helping toparse a continuously moving vector of values depends on how thecoordination between the values changes near a vector of interest. Inparticular, the pause detection routine is very effective in aiding toextract fingerspelled letters from a continuously moving hand statevector due to some important observations about how the coordination ofthe fingers and hand vary near a desired letter. For example, duringfingerspelling, between most letter hand poses the fingers move in acoordinated manner, where the fingers begin moving at the same time,accelerate and decelerate together, and arrive at the final hand pose atthe same time. The few exceptions to this rule are handled on anindividual basis. In the same sense, when one hand pose is reached andthe fingers start toward the next hand pose, relative accelerationsoccur between joints which correspond to curvature of the hand statetrajectory (FIG. 44). These relative accelerations between the jointscause the norm of the hand state velocity vector to unintentionally passthrough a minimum at the letter hand pose. Therefore, in most cases, avelocity minimum corresponds to a fingerspelled letter.

There are, however, exceptions to this rule that must be handledindividually. For example, when fingerspelling "B-O-Y", no intentionalpause need be added to the "O" for a pause to be detected since somepreviously moving joints stop while some non-moving joints begin movingwhen the "O" is reached. These changes in joint coordination produce avelocity minimum. However, if one were to spell, "B-C-O-Y", anintentional pause would be needed to recognize the "C" since it is "onthe way" from a "B" to an "O" and so the joint coordination remainsconstant. This actually doesn't present a problem since most people whowould fingerspell "B-C-O-Y" would also add a short pause to make itvisually clear what the letter was, if it were not clear from context.

Another example is when fingerspelling "A-T" the norm of the hand statevelocity actually passes through a minimum, where there is no letter, onthe way from "A" to "T". In this particular case the "extra" minimum isdue to the fact that in going from "A" to "T" the index finger is liftedover the thumb. As the finger reaches its peak and starts coming down itpasses through a zero velocity point which causes an overall hand statevelocity minimum. If the recognition thresholds are set to a "tight"value, no letter hand pose will be close enough and so the pausedetector won't even be invoked. If the recognition thresholds are set toa "loose" value, then a letter hand pose may be close enough to berecognized, so when the velocity passes through the minimum, anerroneous letter will be recognized. To remedy this situation, a"transition letter" may be trained at the location of each velocityminimum point. These transition letters are hand poses that never showup, i.e., they are never printed nor spoken, but are there as"recognition sinks" to prevent the erroneous recognition of a "real"letter. One possible tedious algorithm to train the transitions lettersis to perform every letter-to-letter transition and let the trainingalgorithm create a new transition letter whenever a velocity minimum isdetected that does not occur "near" one of the trained letters orpreviously defined transition letters.

The instrumented glove and communication system in accordance with theinvention offer distinct advantages over known prior-art communicationsystems. The instrumented glove utilizes linear sensors and isinsensitive to bend radius and joint coupling. Importantly, the glove iseasily and inexpensively manufactured. The use of annealed,high-elongation strain gages in extremely thin and flexible sensorsensures that maximum stress seen by the gages is below acceptablelimits. The glove and communication system provide afingerspelling-to-speech system heretofore unavailable. The systemenables a nonvocal person to "talk" with several hearing or deafindividuals simultaneously. The adaptive letter recognition algorithm isreprogrammable using "soft" decisions. The several peripheral outputdevices enable communication with vocal, nonvocal, deaf and deaf-blindindividuals.

While the invention has been described with reference to specificembodiments, the description is illustrative of the invention and is notto be construed as limiting the invention. For example, the instrumentedglove can be used as an input device in other man-machine environmentssuch as in the control of robotic devices, as a cursor controller, andin artificial limb control, for example. Additionally, the variableresistance strain sensing goniometer sensor in accordance with theinvention can be used in other garments and support structures for useby humans, other animals, and even inanimate objects.

Various modifications and amplifications may occur to those skilled inthe art without departing from the true spirit and scope of theinvention as defined by the appended claims.

We claim:
 1. A sensing assembly comprising:a goniometer comprising aninert thin electrically non-conductive flexible support backing; and avariable resistance strain sensing element affixed to said backing, saidgoniometer having a primary bend axis; means for electrically connectingsaid element; and a guiding element at a position proximal to one endconnected to a position proximal to one end of said goniometer, saidguiding element being flexible about a secondary bend axisnon-intersecting with said primary bend axis and with said secondarybend axis at an angle to said primary bend axis, said guiding elementbeing substantially rigid about an axis colinear with said primary bendaxis, and said goniometer being substantially rigid about an axiscolinear with said secondary bend axis.
 2. A sensing assemblycomprising:a goniometer comprising an inert thin electricallynon-conductive flexible support backing; first and second variableresistance strain sensing elements affixed on opposite sides of saidbacking; and means for electrically connecting each of said elements;means for guiding said goniometer in a predetermined path, said guidingmeans at least partially encompassing said goniometer.
 3. A sensingassembly according to claim 2, wherein said guiding means comprises atubular member through which a portion of said goniometer moves.
 4. Asensing assembly according to claim 3, further comprising a supportmaterial, wherein a portion of said goniometer is affixed to saidsupport material.
 5. A sensing assembly according to claim 4, whereinsaid support material is shaped to fit around a body part.
 6. A hingedsensing assembly comprising:a goniometer comprising an inert thinelectrically non-conductive flexible support backing; a variableresistance strain sensing element affixed on one side of said backing;and means for electrically connecting said element; a support structurehaving first and second members, wherein at least one of said membershas a gap adjacent to the proximal edge of the other member; first andsecond hinged connecting means for connecting said first and secondmembers, wherein said first and second connecting means cooperate toallow said first and second members to move in angular relation to eachother and are separated by said gap; wherein said goniometer extendsover said gap and is affixed to each of said members at opposite ends ofsaid gap.
 7. A hinged assembly according to claim 6, wherein saidgoniometer comprises a second variable resistance strain sensing elementaffixed to the opposite side of said backing and a second means forelectrically connecting said second element.
 8. A sensing system formeasuring the rotation of one site relative to a second site of astructure about an axis, said system comprising:a goniometer comprisingan inert thin electrically non-conductive coiled flexible supportbacking; a variable resistance strain sensing element affixed on oneside of said backing having first and second ends; and means forelectrically connecting said element; means for affixing said first endof said goniometer to said first site; means for affixing said secondend of said goniometer to said second site.
 9. An instrumented glove foruse in a man-machine interface comprising a cover material configured tofit on the hand of a wearer, a plurality of goniometers supported bysaid cover material and each of said goniometers positioned to flex withmovement of the hand, wherein each of said goniometers comprises aflexible, incompressible plastic film and first and second variableresistance strain sensing elements on opposing sides of said film,whereby flexure of each goniometer flexes said first variable resistancestrain sensing element in compression and flexes said second variableresistance strain sensing element in tension, and means electricallyconnected to said plurality of strain sensing elements to the circuitryof the machine system.
 10. An instrumented glove according to claim 9comprising at least one abduction goniometer characterized by beingmounted on edge in a U-shape in proximity to a trapeziometacarpal ormetacarpophalangeal joint; or lying flat on the side of the index orsmall finger in proximity to the metacarpophalangeal joint; or mountedsubstantially straight on edge in line with and on the dorsal side of atrapeziometacarpal or metacarpophalangeal joint.
 11. An instrumentedglove according to claim 10, wherein at least one of a portion of thefingertips, or a portion of the palm of said glove, or a portion of thepalm side of joints is absent or mesh.
 12. An instrumented glove for usein a man-machine interface comprising a cover material configured to fiton the hand of a wearer, comprising a plurality of goniometers supportedby said cover material and positioned to flex with movement of the hand,said cover material including guiding elements in which said goniometersare supported, each goniometer comprising a variable resistance strainsensing element mounted on a flexible support material, and meanselectrically connected to said plurality of strain sensing elements fortransmitting changes in resistance of said plurality of strain sensingelements to circuitry of the machine system.
 13. An instrumented gloveaccording to claim 12, wherein said variable resistance strain sensingelement is free to move at at least one end.
 14. An instrumented gloveaccording to claim 12 comprising at least one abduction goniometercharacterized by being mounted on edge in a U-shape in proximity to atrapeziometacarpal or metacarpophalangeal joint; or lying flat on theside of the index or small finger in proximity to themetacarpophalangeal joint; or mounted substantially straight on edge inline with and on the dorsal side of a trapeziometacarpal ormetacarpophalangeal joint.
 15. An instrumented glove according to claim12, wherein at least one of a portion of the fingertips, or a portion ofthe palm of said glove, or a portion of the palm side of joints isabsent or mesh.
 16. An instrumented glove for use in a man-machineinterface according to claim 12, wherein each goniometer comprises fromone to two variable resistance strain sensing elements, wherein when twosensing elements are used, they are mounted on opposite sides of saidflexible support material, and means electrically connected to saidplurality of strain sensing elements for transmitting changes inresistance of said plurality of strain sensing elements to circuitry ofthe machine system, said plurality of goniometers including goniometersbeing located:over more than one of the trapeziometacarpal,metacarpophalangeal or interphalangeal joints, or in response to therotation or the abduction of said trapeziometacarpal joint, or inresponse to the metacarpocarpal joints of the ring and small finger, orin response to the pitch and yaw of the wrist; or abduction goniometerscharacterized by being mounted on edge in a U-shape in proximity to atrapeziometacarpal or metacarpophalangeal joint, or lying flat on theside of the index or small finger in proximity to themetacarpophalangeal joint; or mounted substantially straight on edge inline with and on the dorsal side of a trapeziometacarpal ormetacarpophalangeal joint.
 17. A sensing assembly comprising:agoniometer comprising an inert thin electrically non-conductive flexiblesupport backing; a variable resistance strain sensing element affixed onsaid backing; and means for electrically connecting said element; meansfor guiding said goniometer in a predetermined path, said guiding meanscomprising a pocket encompassing said goniometer.